SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 1

V-Mail: 3D-Enabled Correspondence

about Spatial Data on (Almost) All Your Devices

Jung Who Nam , Tobias Isenberg , and Daniel F. Keefe , Senior Member, IEEE

Abstract—We present V-Mail, a framework of cross-platform applications, interactive techniques, and communication protocols for

improved multi-person correspondence about spatial 3D datasets. Inspired by the daily use of e-mail, V-Mail seeks to enable a similar

style of rapid, multi-person communication accessible on any device; however, it aims to do this in the new context of spatial 3D

communication, where limited access to 3D graphics hardware typically prevents such communication. The approach integrates visual

data storytelling with data exploration, spatial annotations, and animated transitions. V-Mail “data stories” are exported in a standard

video ﬁle format to establish a common baseline level of access on (almost) any device. The V-Mail framework also includes a series of

complementary client applications and plugins that enable different degrees of story co-authoring and data exploration, adjusted

automatically to match the capabilities of various devices. A lightweight, phone-based V-Mail app makes it possible to annotate data by

adding captions to the video. These spatial annotations are then immediately accessible to team members running high-end 3D graphics

visualization systems that also include a V-Mail client, implemented as a plugin. Results and evaluation from applying V-Mail to assist

communication within an interdisciplinary science team studying Antarctic ice sheets conﬁrm the utility of the asynchronous,

cross-platform collaborative framework while also highlighting some current limitations and opportunities for future work.

Index Terms—Human-computer interaction, visualization of scientiﬁc 3D data, communication, storytelling, immersive analytics.

1 INTRODUCTION

PATIAL data visualization has long been essential for

breakthroughs in medicine, science, engineering, the

humanities, and more (e. g., [1], [6], [18], [41]). Sharing

these visualizations with others, however, remains a major

challenge for many 3D, scientiﬁc, and/or big data visu-

alization applications because they typically require high-

end devices and data processing—the type of computing

infrastructure that is not portable and not accessible to most

non-experts. One strategy for addressing this problem is

to port the exploratory visualization tools scientists use on

their current high-end visualization systems (e. g., advanced

volume rendering) to the commodity devices that people

use regularly, such as through remote rendering [26] or

new web technologies [29], [38]. A fundamental limitation,

however, is that scientiﬁc visualization applications are often

complex, requiring substantial expertise on the data and

on how to use software features. Thus, even when it is

possible for applications to run on commodity devices, there

are still serious challenges to overcome in order to engage

new audiences (e. g., people who may have little incentive

to explore the data) and, often, even our team-science

collaborators, who take on advisory roles in developing

visualization techniques.

In light of this situation, we take a different, comple-

mentary approach. We argue that bringing full-fetched

data exploration capabilities to every device may not be

the most important requirement of high-end collaborative

•

Jung Who Nam and Daniel F. Keefe are with the University of Minnesota,

USA. E-mail: { namxx054 | dfk }@umn.edu .

•

Tobias Isenberg is with Universit´e Paris-Saclay, CNRS, Inria, LISN, France.

E-mail: tobias.isenberg@inria.fr .

Manuscript received December 5, 2022.

visualization for spatial 3D data. Instead, what is fundamen-

tally needed is to improve human-to-human communication

about the data, and the way to do that is to make this

communication work on everyone’s existing devices. This

approach conceptually builds closely on prior work in

storytelling with data visualizations [8], [19], [31], which has

been explored extensively for 2D abstract data but less so for

spatial 3D data. By adopting a keyframe-based storytelling

metaphor [3], [40], we have developed a system that creates

animated data stories from data views captured directly

from exploratory visualizations along with a mechanism of

using an URL of a standard video ﬁle as the base ‘token’

for distributing a data story. By packing the communication

about the data into a standard video format and sending

this critical human-to-human communication rather than the

raw data and visualization application, our system leverages

existing e-mail and social media applications as tools for

distributing stories.

Many forms of storytelling are, however, collaborative

activities by nature [20], [23], [42], and people often take on

different roles in this process [7], [11], [23]. We, thus, also

support tasks that data story audiences may carry out after

an initial viewing of a story such as providing feedback to

the original author, verifying claims made in the story, or

even adding new content to the story.

To turn data storytelling into a collaborative and iterative

process, we extend our system and introduce the V(isualiza-

tion)-Mail framework. The key approach is ﬂexible export

of data story where a single V-Mail story token can trigger

experiences with different storytelling ﬁdelities, based on

the absence or presence of dedicated software (Fig. 1). By

having options from simply viewing the story on a video

player to opening it within specialized visualization software

to explore and annotate the 3D data, it is possible to cover

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2

Fig. 1. Three working platforms that demonstrate different storytelling ﬁdelities of V-Mail for analyzing Antarctic ice-sheet data: (right) anyone with a

video player can at least passively view the story, including annotations made by others; (left) in the highest-ﬁdelity case, the story unlocks data on a

V-Mail server than can be loaded via a plugin for desktop-based visualization applications, where users can explore and annotate the 3D data more

deeply; (middle) the mobile client works as a custom video player with mechanisms for interactively viewing the story and adding annotations.

a wide range of use-cases for collaborative discussion and

3D data analysis, including making it possible for users with

different tasks and levels of familiarity with visualization

applications to work in unison.

We also present several extensions to this core idea to

further enhance 3D-speciﬁc data storytelling. For example,

since understanding the spatial context for speciﬁc data

views is often critically important, the views are represented

as 3D frames ﬂoating in the same space as the data visu-

alization, enabling the audience to see them from various

vintage points. In addition, to increase clarity, we employ a

staged animation when the transition between data views

would require changes to several visualization parameters,

for example, when new data ﬁlters or color maps have been

applied.

In summary, the main contributions of this article are:

•

the design of a novel visualization framework to fa-

cilitate 3D-enabled multi-person correspondence about

spatial datasets for team science,

•

an extension of keyframe-based storytelling metaphors

to support co-authorship, treating the created stories as

collaborative documents authored asynchronously,

•

the design of a data story server and communication

protocol for using V-Mail across an ecosystem of hard-

ware platforms that seamlessly adapt the ﬁdelity of the

experience to the hardware, and

•

user feedback and lessons learned from a ﬁrst evaluation

of using V-Mail in a real team-science setting where team

members utilized all V-Mail client platforms we created

(desktop app, mobile app, and native video players on

multiple mobile and web-based devices).

2 RELATED WORK

V-Mail relates to research on interactive tools, techniques, and

formats for storytelling with data visualization and builds

upon multiple topics (e. g., animation and annotation, inter-

activity, co-authorship and collaboration) that have become

consistent themes in this research. We note that, although

storytelling techniques have been used for presenting both

spatial [27], [28], [36], [40] and non-spatial [4], [10], [22], [23],

[32], [34] data, recent work has focused more consistently on

non-spatial data. V-Mail, therefore, builds upon a trend of

customizing storytelling support tools for speciﬁc types of

data (e. g., [3], [25], [35], [40]), but does so for the speciﬁc case

of spatial 3D data, as often encountered in domains such as

physics, astronomy, biology, and medicine.

2.1 Co-Authoring and Interacting with Stories

Many forms of storytelling are, by nature, collaborative

activities [20], [23], [42] and people often take on different

roles in this process [7], [11], [23]. In a recent survey, Tong

et al. [36] describe authoring tools for parallel storytelling,

where multiple authors work simultaneously to create the

ﬁnal visualization content. Like V-Mail, this type of story-

telling blurs the lines between storyteller and audience; the

audience partakes in the storytelling and the initial storyteller

is also an audience member. In sense.us [13], for example,

users communicate collaboratively using annotations and

comments posted online on web-based data representations.

The audience also partakes in the storytelling process in

GeoTime [9]. In an accompanying text editor, authors write

text-based data stories using bookmarked data views as

references to claims. By scrolling the editor ﬁlled with

the links, the audience has the ability not only to see the

message with text descriptions but also to ﬂip through

saved visualization states for further exploration. Further,

by modifying text in the editor, the audience can add new

content or even new data views found in their own data

analysis sessions.

V-Mail is similar to GeoTime in that multiple users

collaborate on visualization presentations. Unlike GeoTime,

however, V-Mail employs several simple yet effective tech-

niques to convey different users’ contributions. For example,

as in the work by Heer, Viegas, et al. [13], [39], each comment

shows the author of the edit. Further, V-Mail incorporates

lower-ﬁdelity options for viewing and collaborating with

others on low-cost devices such as mobile phones. By sharing

a visualization state via a URL [13], [39], V-Mails can be refer-

enced in other communication media such as personal blogs.

Similarly, by using video ﬁles as the means of distribution

[14], V-Mails can be widely distributed, including through

social media. V-Mails provide users with different options for

consuming data stories, from passive viewing to interacting

with them, as there is no one correct level of interactivity

on the spectrum for constructing a narrative visualization

[11], [28], [34]. While interaction can distract viewers from

an initial story arc [20], the ability to interactively view data

stories is typically seen as giving the audience a feeling of

involvement and lends credibility [21], [27], [28], [40], [42].

Existing web-based systems that rely on remote rendering

[26] or new web technologies [29], [38] can also allow people

to carry out visualization tasks on the commodity devices

they use regularly. These systems, however, often target data

exploration tasks, whereas we focus on data communication

with the support for collaborative annotation and storytelling.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 3

Also, our ﬁrst prototype provides storytelling techniques on

top of visual outputs from exploratory visualization systems,

so the V-Mail concept could be applied to visualization

software built as standalone applications.

2.2 Annotating Spatial 3D Data

When compared to creating data stories for 2D or non-

spatial data, authoring stories about 3D spatial data includes

another special consideration in that it requires connecting

text, speech, drawings, or other input from authors to speciﬁc

spatial views or data elements. These software features are

often described as 3D annotations, which can be tied directly

to data objects [37], [40]. Alternatively, graphical annotations

can be drawn in ‘screen-space’ [3], [13], [25], [37]. Without

the need to explicitly reference data elements, this ‘screen-

space’ drawing with ‘free-form’ annotations enables users to

be more expressive [13] and even to draw in ‘empty space,’

which can be useful in sparse 3D scenes [37]. However,

free-form ‘screen-space’ annotations are typically not useful

when viewed from other vantage points, as the spatial

context is only valid from the exact viewpoint where the

annotations were made. Some form of navigation between

annotations is needed along with an ability to see what

viewpoints include annotations. V-Mail thus implements

screen-space annotations in the style of Boom Chameleon

[37]. After creating an annotation from one viewpoint, we

add a snapshot of the annotated view to the scene as a 3D

widget (view frustum with a textured ﬁlm plane) that can be

seen from other vantage points and returned to later.

2.3 Arranging and Transitioning Between Multiple Views

Many data storytelling systems provide multiple comple-

mentary data views, and they are considered critical for

3D data storytelling [3], [36], [40]. One reason is that

understanding the complex spatial relationships in 3D data

often requires multiple and/or animated perspectives [20],

[24], [28], [34]. By smoothly transitioning between these

perspectives, well-designed animations can allow the audi-

ence to build a congruent mental model of the spatial data

[10]. Different mechanisms exist for authoring animations

for data storytelling. These can be as simple as recording a

sequence of interactions performed by the storyteller [40]

or providing animation templates to assist users in creating

a ﬁnal animation [3]. Authoring a meaningful animation,

however, can be complicated. Automated techniques are thus

also useful, especially in time-critical scenarios such as urgent

care in a hospital [14] and even for science dissemination [21].

V-Mail, adopts a semi-automatic approach, similar to that of

Lidal et al. [25]; the animations are created by interpolating

the values the storyteller has captured/created in discrete

states. We extend this concept by utilizing a staged animation

[3], [12] to increase clarity when the transition would require

changes to several parameters at once. Additionally, V-Mail

includes a linked 2D storyboard view to deﬁne the temporal

context for each key state and to re-sequence the story if

desired. This feature is motivated by prior work that created

guided 3D visualizations by capturing and ordering multiple

views and found that breaking the presentation into steps

can improve comprehensibility [40].

3 D ESIGN CRITERIA AND CORE DE SIGN CONCEPTS

Based on this discussion of existing means for communicat-

ing about or using spatial 3D data and the challenges we

outlined in the previous sections, we now translate these

points into speciﬁc design criteria and introduce the design

concepts of V-Mail that guided us toward our vision.

3.1 Design Criteria

Our overall approach is driven by today’s huge spectrum

of data communication possibilities, yet with no solution

that matches our vision. On one end of the spectrum,

people use e-mails enriched with screenshots from general-

purpose visualization tools such as ParaView [2]. This has

the advantage that one does not have to worry about the

device of the addressee, yet messages are limited to texts

with few images, with little expressiveness. Video exports can

also be used to showcase visual representations with higher

ﬁdelity and in a continuous fashion, yet this only supports

one-way communication. Moreover, communication features

such as drawing and commenting are typically missing in

e-mail based communication. The other end of the spectrum

is marked by storytelling tools, such as that by Wohlfart

and Hauser [40], in which recipients can load a shared

visualization not only to view but also to explore. Unfor-

tunately, such tools require collaborators to have a similar

hardware and software setup and often hours of training

to use the visualization application. As argued previously,

this often makes them accessible only to experts. Our key

goal is thus to facilitate communication about spatial 3D data

without having to worry about recipients’ devices and skill

levels, yet with an ease of communication, responding, and

commenting that approaches that of e-mail. To achieve this

vision, we formulated the following design criteria, which

are closely informed by ongoing data visualization research

on storytelling, communication, and collaboration:

As audiences have different goals and needs,

we need to provide them with different options for

consuming the presented material, e. g., with different

levels of interaction and detail [11], [23], [42] and

we want to support a range of devices to view messages

and provide feedback so users can pick a visualization

suitable for their tasks and current situations [15].

As there can be multiple ways of interpreting results,

messages should capture the thought process that lead

to a conclusion so others can make informed decisions

[25], [30], [40]

and, when going through the steps of presenting spatial

data, it is important to a method should be provided for

the audience to maintain spatial correspondence as the

presentation proceeds and to understand where, spatially

within the data, each step takes place [21], [28], [36], [37],

presenters should thus have various options to express

their ideas and thoughts and to draw the audience’s

attention to particular viewpoints and data features [25],

and

the audience should have the ability to add to and edit

the content so that groups can exchange ideas back

and forth iteratively as they work collaboratively to

understand and interpret the impact of the data [23].

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 4

3.2 V-Mail Core Metaphor and Design Concepts

The core metaphor behind V-Mail is that of a book—

speciﬁcally, we recall children’s’ picture books that include

text and imagery on each page and are often organized so

that each page functions as a mini “episode” of the story. A

child looks at a page in a book and reads the text at their

own pace to understand the setting and characters pictured.

Then, to advance, the child ﬂips to the next page to continue

to the next episode. Books for adult readers follow a similar

progression of steps, but this is often arranged at the level of

chapters. The relationship of one chapter to the next may not

be immediately apparent, but the perspective provided by

each chapter is, ultimately, critical to the interpretation of the

whole book. Books of all forms naturally lend themselves to

a variety of annotations, from hand-drawn illustrations to

remarks scribbled in the margins.

In V-Mail’s approach to storytelling with spatial 3D data,

a ‘page’ is a

view of the 3D dataset

that shows a particular

situation. Annotations can be provided by

sketching

onto

the 2D projection of the 3D view or by attaching textual

annotations to the view. ‘Flipping to the next page’

animates

from one 3D view to the next

, allowing the viewer to follow

along while maintaining spatial context. Several page ﬂips

can be connected into ‘

chapters

’, but the transition from

the end of one chapter to the beginning of the next can be

more abrupt (i.e., without an animated transition). As in the

children’s picture book, it is possible to animate

forward and

backward

through the data story, with

different playback

modes

to control the time with precision or to quickly skip

to a new chapter. With this approach, complex messages are

broken down into more digestible, visual, and expressive

forms (C3–C5).

The V-Mail frameworks requires just three additional con-

cepts to be added to this core “book metaphor” functionality:

Floating 3D pages

—When presenting visualization re-

sults with a set of captured data views, we often think of

a panel listing the views; e. g., thumbnails on the viewer in

AniViz [3]. V-Mail includes such a panel, but in addition

to this panel, the data views are also included directly

within the spatial visualization of the underlying 3D data.

Together with the smooth transitions between the data views

achieved with animations, these ﬂoating pages assist users

in understand the spatial correspondences between views as

they progress through a data story (C4).

Living documents

—To remove the one-way character

of existing 3D data communication and turn the data

communication into a collaborative and iterative process

accessible to non-experts, V-Mail treats the data stories as

living documents (C1, C6). The framework makes it possible

for users to explore a V-Mail, provide annotations, and extend

stories in a way that scales with the capabilities of the device

being used.

Communication via a

lightweight token

—To realize this

device ﬂexibility (C2) and thus be able to engage a broad

audience (C1), V-Mail does not rely on dedicated rendering

support. Instead, stories are shared as a video URL—online

token that can be interpreted based on the used device

and the absence or presence of dedicated software. The

video-based, low-ﬁdelity version of the data story makes it

possible for users to open it on a mobile phone with a video

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 2. The V-Mail desktop client allows users to create a story directly

from a visualization application: (a) the top portion is the visualization

application; (b) the storyboard panel is placed on the bottom; (c) captured

data views in a story; (d) the timeline interface provides playback options

and indicates the current time in a story; (e) the drop-down menu provides

options for opening, uploading, and sharing a story; (f) the comment panel

shows comments on the current data view.

player, for instance while traveling. However, this lowest

level of ﬁdelity supports only simple viewing options (e.g.,

the web client in Table 1). A range of higher-ﬁdelity versions

that require dedicated software offer additional exploration

and annotation capabilities. On a high-end desktop, users

can freely explore the data depicted in the story and add

additional pages or chapters (e.g., the desktop client in

Table 1). A middle-ﬁdelity version only supports exploration

at certain view locations but with fairly advanced annotation

capabilities and story navigation features (e.g., the mobile

client in Table 1). To make it easy to manage such device

ﬂexibility, V-Mail associates all story data with the video, and

shares it as a URL token that is opened appropriately by the

different devices.

4 DETAILED DESIGN AND IMPLEMENTATION

We now describe the visual components, interaction tech-

niques, and distribution mechanisms required to realize this

vision.

4.1 Visual Components

A V-Mail story is composed of data views with drawings

and comments from different users. We add animations

in-between the data views to convey their spatial correspon-

dences in a continuous fashion. Our V-Mail desktop client is

one example of a V-Mail client (Section 4.3.3) and is a plugin

to an exploratory visualization application [17]. The plugin

is comprised of three visual components: (1) the storyboard,

(2) ﬂoating 3D data views, and (3) animated-transitions.

4.1.1 The Storyboard with the Comment Panel

This ﬁrst visual component shows an overview of a data

story and the input from contributors (Fig. 2(b)). Thumbnails

of data views provide a quick glance at important points in

a story and its presentation order (Fig. 2(c)). We also overlay

drawings on the thumbnails to indicate visual highlights from

users. Sometimes it may be more appropriate to textually

describe what is shown in a shared data view; e. g., for

writing a case number in a data ensemble or the name of a

rendering algorithm. The comment panel provides an interface

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 5

(a) (b)

Fig. 3. Example usages of 3D data views in data communication: (a) the

user talks about data features focused in the views; (b) a bird’s eye view

shows the locations being discussed in the data.

for leaving such notes and viewing comments left by others

(Fig. 2(f)); each comment includes the author’s name.

4.1.2 Floating 3D Data Views

This second visual component shows the user how the

story is spatially located in the 3D data space. As shown

in Fig. 3 and inspired by Tsang et al. [37], captured data

views are rendered in the same space as the underlying data

visualization so that they are visible from other vantage

points. If a data view is captured close to a feature of

interest, the user can zoom out and see a representation

of that viewpoint in the context of a wider angle view of

the data. By making this spatial information visually explicit,

users can also visually reference other data views from the

current view: a shot can show adjoining data views to convey

their spatial relationship, before transitioning to each view

(Fig. 3(a)). A spatial overview of all of the activities in a data

analysis session can also be easily created by showing the

data views scattered around visualization, from a bird’s-eye

perspective (Fig. 3(b)).

4.1.3 Animated Transitions

Animated transitions help to convey the spatial and contex-

tual correspondences between data views in a story. Instead

of jumping from one data view to the next, we use animated

transitions that the audience can gradually adapt to the visual

changes in different data views. The animated transitions are

created automatically whenever a new data view is added

to minimize the time and cognitive effort needed to create a

V-Mail story.

Four types of animated transitions (Fig. 4) can be used,

depending on the differences between the viewpoints and

visualization states in the adjoining data views. If only the

camera location changes, the animation (1) translates the

camera from a starting location to the next location.

When

only the visualization state has changed, the animation (2)

gradually changes the values of visualization parameters

from the previous state to the next by interpolating the

values of just the parameters that have changed. Users

may stop the transition to explore the visualization in an

intermediate frame, but this exploration is only possible

when the set of visualization parameters of two states are

the same and the parameters can be changed continuously

(e. g., changing a slicing plane in volume rendering). In

cases where there are no obvious intermediate frames (e. g.,

changing a visualization type or dataset) we (3) use an image-

based blending approach. This blends two images where

the screenshot from the previous data view fades-out and

gradually fades-in into that of the next data view (see the

second row of Fig. 4). When both camera location and state

change, we (4) break the transition into two stages. First the

camera moves to the next location. Then, the visualization

state gradually changes to that of the new data view (see the

last row of Fig. 4).

In our prototype, a linear interpolation is used; however, other

motions that optimize camera motion to support speciﬁc goals could

be useful. We thus implemented the animated transitions as abstract

methods to enable future developers to override default animations

with alternatives.

cam. vis. type animated transitions

6= = 1

=======================================================⇒

move to the next location

= 6=

=======================================================⇒

interpolate to the next visualization state by parameter (type 2) or through blending (type 3)

6= 6= 4

==========================⇒

move to the next location

==========================⇒

interpolate to the next visualization state

Fig. 4. Different animations to transition from one data view to another. Each type shows the changes in-between two adjacent data views.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 6

Fig. 5. The user sketches and leaves comments on the current view in

the annotation mode. A drawing pallet is for specifying the color and the

size of a brush. Clicking the toggle on the top of the storyboard switches

the current mode to the exploration mode.

4.2 Basic Story Interaction

We introduce several interaction techniques to create, view,

and explore V-Mails.

4.2.1 Creating a story and annotating data views

We begin creating a story with an empty storyboard. Users

can navigate in the data visualization application (Fig. 2(a))

using typical 3D view manipulation techniques, and then

add a data view to the storyboard via a dedicated button

(“+” in Fig. 2(b)). We then capture the view’s thumbnail and

insert it as a reference into the storyboard, also extending

the timeline (Fig. 2(d)) appropriately (for

≥

2 thumbnails)

to account for the needed transition animations. Users can

also re-arrange the data view sequence using simple drag-

and-drop interaction, and they can remove data views as

desired.

To be able to tell a story effectively or to respond to one,

it is also often useful to sketch directly on a view to direct

someone’s attention to particular features, which we support

on captured data views (Fig. 5). To avoid conﬂicts between

sketching and view manipulation, we differentiate explicitly

between these two modes and users can switch between

them using two buttons at the bottom of the data view. In

addition to the sketches, longer textual comments can be

typed and associated with the data views (Fig. 2(f)).

4.2.2 Uploading a story

When ready to share the V-Mail, users upload the story to a

server. Instead of uploading a story to the server whenever

there is a change, we initiate this upload process explicitly

(via an entry in the drop-down menu). We chose this design

to enable the user to ideate and draft a story before sharing

it to others, similar to drafting an e-mail before sending it.

4.2.3 Viewing an animated data story

As a sequence of data views with animated transitions, a

V-Mail story is akin to a video with chapters. We thus borrow

different playback modes from traditional video players and

adjust them to our application (Fig. 6).

As in normal video players, playback continues until

the end of the story (top row in Fig. 6). However, different

from normal videos, the V-Mail player pauses brieﬂy at each

‘chapter mark’ to devote additional time to and indicate the

Data View #1 Data View #2 Data View #3

Fig. 6. Playback options & corresponding buttons. Top: plays the anima-

tion from start to ﬁnish, or in reverse; middle: plays it until it reaches the

next/previous data view; bottom: jumps to the next/previous data view.

(a) (b)

Fig. 7. The data visualization application and the storyboard are visually

and interactively linked. (a) Hovering over a 3D data view frame in the

3D view (the center one here) highlights the frame and its corresponding

thumbnail on the storyboard; (b) clicking on a data view frame in the 3D

view makes the storyboard jump to this data view.

signiﬁcance of these key perspectives. To further emphasize

the data views, the player also includes mode where playback

stops completely at each data view so users may read

annotations in detail (second row in Fig. 6) and respond

to them—similar to the playback style used by Wohlfahrt

and Hauser [40]. Finally, for faster navigation, the player also

supports jumping between data views without animations

(bottom row in Fig. 6).

Another aspect of data stories that

is different from normal videos is that it can be useful to

view data stories, especially the transitions between views,

in reverse. Thus, all three of the play modes include support

for reverse playback.

To allow users to explore V-Mails in a more ﬁne-grained

manner, similar to video players, we also provide a time-

line slider (Fig. 2(d)). To emphasize the aspect of chapter

transitions we constrain the timeline dragging to between

two neighboring data views, so that it is easy for a user to

focus on the visualization view while controlling the timeline

with a mouse. This constraint is reset on each mouse down

(for exploring a new transition) and can be removed with a

modiﬁer key.

4.2.4 Linking free exploration and storyboard timeline

Again different from traditional video players, the interaction

in a data story is not restricted to just playing back a pre-

created video stream. Instead, a user may exploring the

3D data visualization freely, potentially to capture slightly

different shots or to better understand a feature in the data.

After such a free exploration (and if no new shot was added

to a timeline), the viewer may want to transition back to

the timeline. We support both of these types of transitions

by keeping the 3D visualization view and the timeline

exploration linked and use animation transitions from the

set described in Section 4.1.3. Each time a user switches

The desktop client does not have these skip buttons because the

skipping navigation can be done by clicking thumbnails (Fig. 2(c)). In

the mobile client, however, we place the buttons in the bottom corners

of the display to provide easier access (Fig. 9).

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 7

(a)

(b) (c)

Fig. 8. A story URL works as a lightweight token for sharing and opening

the story. (a) our V-Mail clients show several stories with their URL;

(b) after a story’s upload, the client shows the URL; (c) opening the URL

takes the user to a webpage where the story video can be played and

the story can be opened in a dedicated client via a button (bottom).

from timeline to 3D view, we store the current position in

the timeline to be able to transition back. In the opposite

direction, we create a transition from a current state in the

3D viewer to the last active one. We also allow users to

jump to stored data views, not only by interacting with the

storyboard view, but also by clicking on the respective data

view frames in the 3D view (Fig. 7).

4.3 Cross-platform Collaboration Mechanisms

To truly support our cross-device collaboration vision (C2)

we cannot rely on all users having access to a high-end PC,

which may be required for the style of interection described

this far. We thus now describe how users respond to V-

Mails on different devices and introduce the platform-speciﬁc

prototypes (i. e., V-Mail clients) that we have developed so

far.

4.3.1 Flexible export of data story

As we want to support a wide ecology of devices, we have

to ensure that they can show a V-Mail without advanced

rendering hardware and even potentially without a dedicated

client. As a most basic representation we thus rely on a

traditional video that we generate from the data story, just as

if one would have used a normal playback to watch it. Any

device that supports video ﬁle playback (we use MP4 ﬁles)

can thus show this most basic V-Mail format—including

virtually all mobile phones, tablets, and laptops. Devices or

players that also support video subtitles embedded in video

ﬁles can also show the textual comments. Similarly, we also

embed chapter marks for those video players that support it.

If a person receives V-Mails on a more regular basis, they

can decide to install a dedicated tool that can enable them to

further engage with V-Mail. To support additional features,

we export a collection of ﬁles for each story in addition to the

video ﬁle (Table 2 in Section 5). In this case we provide access

to the dedicated play modes we described in Section 4.2.3,

and for this purpose we also include a video ﬁle exported

in the reverse direction (video backward.mp4). To allow users

to adjust or add to the graphical and textual comments, we

save both types as separate ﬁles (ann i.png, msg.json), that

can be updated by the dedicated player after a change. In

addition, we provide some simple ‘3D exploration’ on low-

power devices. As we cannot rely on rendering hardware, we

use 360

images that we create during story export for each

of the data views (img360 i.png). We also store all data views,

camera locations, and visualization states in the msg.json

(a)

(b)

(c)

Fig. 9. We created the V-Mail mobile client (a) as a custom video player

with visual components and mechanisms for interactively viewing and

editing a story; clicking on the top-left share button in the viewer copies

the story’s URL and shows messaging apps to share the URL with (b);

when clicking an e-mail app, we create a message with the name of the

story as a title and the story’s URL pasted into the main body (c).

ﬁle for users with sufﬁcient high-end hardware and the full

visualization software. Only with this data ix ig possible

to adjust the data views, add new ones, rearrange their

sequence, and re-export an updated story.

4.3.2 URL as a lightweight token for story sharing

While we export a collection of ﬁles for each story (Table 2),

we do not want to bother users with having to manage and

send these ﬁles. We thus manage the ﬁle collection on a

dedicated server (that only has to provide ﬁle management

and remote access) and instead share a story in form of

a URL that points to the repository. An essential beneﬁt

of using a URL is that this allows users to not only link

to a story in an e-mail but also to share a story on social

media. We create this URL after each export (Fig. 8(b)). When

opened in a web browser, the URL simply shows a video

player (Fig. 8(c)). On devices with access to our dedicated

clients the website, the URL provides a means to open the

story in the dedicated software. Alternatively, the clients also

provide a list of stories stored on the server (Fig. 8(a)). In

both cases, the clients then access the ﬁles needed for their

speciﬁc capabilities and provide the respective interaction

tools, as we detail next.

4.3.3 A device ecology for collaborating on data stories

We implemented three speciﬁc clients as a proof of concept

and to mirror the data levels we described in Section 4.3.1.

In Fig. 1 we show photos of these solutions and in Table 1

we compare them by the features they support.

As just described, our most basic

web client

relies on

the video player available on virtually all connected devices

(Fig. 8(c)) to facilitate passive storytelling [40]. Depending

on the functionality of this player, one may only perceive the

story as a simple playback with graphical annotations for the

data views. If the player supports it, however, one may also

jump between chapters and/or see the textual comments

that are available as video subtitles.

The web client can be accessed on virtually any connected

device and uses the device’s built-in video playback. With

it we can thus reach completely new audiences with data

stories. Even if someone never used a data story before and

thus does not have or is not willing to install dedicated

software, they can still view them. Depending on the media

channel used to share the URL, they can also respond to a

story such as by commenting in a separate e-mail. Moreover,

this form of consuming also allows users to explore the story

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 8

TABLE 1

Availability and ﬁdelity of operations that facilitate collaboration around shared stories and their prerequisites, by client/platform.

web client: traditional video player mobile client: custom video player desktop client: plugin to vis. tool

— interactive storytelling —

story navigation

playback and timeline controls with-

out pausing points

playback and timeline controls with pausing points,

storytelling affordance that distinguishes story nodes and transitions

exploration no support

(i. e., passive story-

telling [40])

partially

(360

views; i. e., storytelling

with interactive approval [40])

full support

(all four modes of author-

and reader-driven storytelling [40])

— collaboration around shared stories —

view comments partially (if player shows subtitles) yes yes

view annotations yes yes yes

leave comments no support yes yes

add annotations no support yes yes

extend story no support no support yes (add, remove, rearrange story)

— software and hardware prerequisites —

software needs none dedicated app special visualization software

hardware needs virtually none virtually none high-end GPU & CPU processing

on their own ﬁrst [23], before, potentially, adding to it and

working collaboratively with other authors.

To explore the situation where users are willing to install

a dedicated tool, such as a

dedicated mobile app

on their

phone or tablet, we developed an Android V-Mail client app

(Fig. 9), which makes possible a higher ﬁdelity of interaction

for both phones and tablets. On top of playing back the video

of the data stories, the app also allows users to make use of

our advanced playback modes, add new annotations (both

graphical and textual), and have limited 3D exploration via

the pre-computed 360

images for each data view. As it is

likely used on small-screen devices, the app does not support

changing the chapter order, and instead of using thumbnails

for the data views in the storyboard, the app places crumbs

on the timeline to provide visual and navigational support

to users.

The mobile app facilitates a much richer form of col-

laboration by means of data stories than the web client,

including being able to directly comment within the viewing

environment and some 3D exploration. However, since this

functionality relies only on the built-in video playback avail-

able in virtually all devices, the processing power required

is essentially the same as the web client. Also, because it

runs on mobile devices, it inherently allows people to engage

with data stories like they do with other online media, for

example, during a break, at home, or while traveling. In fact,

the URL format makes it possible for people to share V-Mails

on a variety of communication platforms, from e-mail to

messaging apps, already installed on their phones.

For full-ﬁdelity interaction, however, we created a

desk-

top V-Mail plugin

to a data visualization application [17]

for both Windows and MacOS. This setup provides the

functionality described in Section 4.2. It requires a high-

performance computer, but it provides full ﬂexibility in

data exploration, story creation, and editing. At least one

desktop client like this is needed to create an initial sharable

data story, and this high-end client also makes sure that

any updates from mobile app users get re-embedded into

the video exports. This client also supports creating story

branches with new sharable URLs.

Video Renderer

360° Images Viewer

Exploratory

Visualization

Software

Storyline Edits

Annotations

Story Navigation

Desktop Plugin

File

Storage

V-Mail Server

Data

Annotations

Story Navigation

Mobile Client

HTML <video>

Web Client

Fig. 10. System architecture. V-Mail is composed of a server and three

visualization clients. The desktop plugin is connected to an exploratory

visualization software. The mobile and web clients support a portion of

data storytelling activities around visual outputs stored in the server.

5 SYSTEM AND IMPLEMENTATION NOTES

Our current V-Mail implementation comprises three clients

(Fig. 10). We built the desktop plugin on top of an exploratory

visualization software; it provides all the storytelling capa-

bilities. When the user of the plugin uploads a story, ﬁles

shown in Table 2 are generated and uploaded to the V-Mail

server. One of these ﬁles is a video, and the mobile and web

clients work with this video ﬁle. The mobile client is a special

video player for data videos with the dedicated play modes

we described in Section 4.2.3 and annotation features. The

web client relies on a video player within a browser.

5.1 V-Mail server and general operation

The V-Mail server comprises a database and a ﬁle storage.

The purpose of integrating the database is to check available

stories without opening ﬁles in the storage. It has a single

table; each row represents a V-Mail story, and the columns

are: an unique ID of the story, a name given by the user who

created it, and last times modiﬁed by each client. The ﬁle

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 9

TABLE 2

Files in a V-Mail story folder

File Use by Description

msg.json Desktop, Mobile Contains information about camera views, visualization states, and comments on data views.

video.mp4 Mobile, Web Shows the animation as a video ﬁle.

video backward.mp4 Mobile Shows the animation in a reverse direction, and is used to provide a smooth backward play.

— per data view —

img i.png Desktop A snapshot of the current view, that is shown as a thumbnail for the data view

nextImg i.png Desktop

A snapshot captured from a camera location of the next data view, that is used to support the

fourth type of animated transition

annImg i.png Desktop, Mobile An image of drawings

img360 i.png Mobile A 360° image of the view, that is used to provide limited 3D exploration

— for the web-client —

index.html Web The default page

subtitle.vtt Web Shows comments as a subtitle for the video.

playlist.m3u Web Provides data view portions as chapters in the video.

storage contains additional ﬁles for V-Mail stories (Table 2).

Files for each story are saved in separate folders, and names

of the folders are the unique IDs listed in the database.

When a user opens the desktop or mobile client, the client

accesses available stories from the database and shows these

as a list (Fig. 8(a)). When the user clicks a button to open

a speciﬁc story, ﬁles used by the current client (“Use by”

column in Table 2) are downloaded and loaded. When the

user uploads a story from the desktop plugin, all the ﬁles are

generated and uploaded to the server. The mobile client only

modiﬁes msg.json and annImg i.png ﬁles.

When the user adds annotations on the mobile client,

we need to re-create video.mp4 and video backward.mp4 to

reﬂect the changes. For this purpose we integrated another

server-based application into the V-Mail infrastructure. This

application shares much functionality with the desktop client

and checks the database every minute for the changes. If it

ﬁnds any, it recreates the video ﬁles and uploads these to the

server. For this check we compare the last modiﬁed time by

a mobile client with that of a desktop in the database.

The mobile client plays video.mp4 or video backward.mp4

depending on the current play direction. When the user

clicks a backward playback button (Fig. 6) while watching

a video in a forward direction, the client skips time in

video backward.mp4 to match that in video.mp4 and plays

video backward.mp4. The same happens for video.mp4 when

the user clicks a forward playback button.

We built the desktop and mobile clients with Unity3D,

and the web client is a web page with a HTML video element.

We realized the database with MySQL, and wrote the

server-side scripts for accessing the database and download-

ing/uploading ﬁles in PHP. We used an FFmpeg executable

ﬁle for creating a video ﬁle from a series of images captured

from an exploratory visualization application [17].

5.2 Desktop and mobile clients details

We built the desktop and mobile clients using C# in Unity3D.

We handle storytelling and collaborative activities in V-

Mail with a StoryManager (Fig. 11). In addition to providing

features for loading and uploading a story to the server, the

StoryManager has three components for viewing and editing

a story. A Message manages content in the current story and

has an ordered list of Page(s). Each Page contains view and

Page

Contains infomration

about a single data view

Message

MessagePlayer

(Desktop only)

Computes the current

Transition based on the

time and Message

<<interface>>

IViewer

void SetState(Page page)

void SetState(Transition t)

MessagePlayer

(Mobile only)

Similar to the desktop

MessagePlayer yet the

time relies on a video

Exploratory Vis. Software

(Desktop only)

360° Images Viewer

(Mobile only)

Video Player

(Mobile only)

Transition

Contains two adjoining

Pages and the current

transition amount (0~1)

CommentViewer

AnnotationViewer

SphericalImageViewer

(Mobile only)

VisualizationViewer

(Desktop only)

1..*

Fig. 11. Three components of viewing/editing a story in StoryManager

visualization state information captured from an underlying

visualization application. It also contains comments and

sketches drawn by users. The MessagePlayer updates the

story time depending on the current playback option (Fig. 6).

It also provides methods for accessing the current Page and

Transition in a story. A Transition is composed of two Pages and

a transitioning amount. Viewers are visual components that

show the current Message and provide a means for editing

it. CommentViewer updates the comment panel (Fig. 2), and

AnnotationViewer shows drawings on the render view. The

VisualizationViewer updates an underlying visualization. If the

current time is on a page, the viewer updates the visualiza-

tion from the current Page accessed from the MessagePlayer. If

on a transition portion, the viewer interpolates between two

Pages from the Transition. As the mobile client plays video

ﬁles, the StoryManager in the client is different from that of

the desktop plugin. The VisualizationViewer is removed as the

client does not rely on an underlying visualization, and we

add a SphericalImageViewer to provide limited 3D exploration.

The MessagePlayer updates the timeline slider based on the

current time in the video, and skips time upon clicks on the

corresponding buttons or a speciﬁc point on the slider.

6 APPLICATION CASE AND EVALUATION

To evaluate the effectiveness of our concept for team collabo-

ration, we recruited members of a leading, interdisciplinary

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 10

climate science research team

to integrate V-Mail into their

normal workﬂow over a two-week period. This section

reports on training, observations, results, user feedback, and

other lessons learned from this real-world application. We

also showed the tool to members of a

neuroscience research

team

, and report lessons and user feedback from their initial

encounter with our tool in Section 6.3. Unlike the climate

science team, they decided not to be part of a longitudinal

study. In the section, we also report limitations and areas

could be addressed in future work.

6.1 Methodology

The interdisciplinary climate science research team uses a

combination of tools, including some of the world’s most

advanced high-performance computing (i.e., supercomputer)

simulations, to study complex interactions of Antarctic water

masses that may explain the global rise in sea temperature.

The team includes oceanographers, geologists, computer

scientists, a visual artist, and a science writer. Like many

research teams today, they place an increasing emphasis

on improved communication with the public. The artist and

writer have played leading roles in these efforts, and the team

has produced ﬁve short videos describing the science in the

past seven years, all of which were national competition

ﬁnalists (two won awards as best submissions). One of their

recent videos was created with the Unity-based Artifact-

Based Rendering (ABR) engine [17]—the same engine used

for our ﬁrst implementation of a V-Mail plugin.

Since one of our goals was to understand how V-Mail

might integrate with whatever data communication work

is at hand in a real, research-active science team, we asked

the team to use V-Mail for whatever tasks they thought

made sense during the two-week trial we had agreed upon.

The team decided that the most interesting communication

task

at hand was to develop one or more complementary

videos to go deeper into certain aspects of the science and

provide useful material for the scientists to use in their

presentations. Thus, exploring how V-Mail might be utilized

to support this work became the primary task of the V-

Mail trial, and the

two primary users

became the visual

artist, a professional sculptor and painter for more than 30

years whose collaborations with scientists led her to data

visualization, and the science writer, a Ph.D. student in

cultural geography. During the ﬁrst few days of the two-

week period, we met individually with the artist and science

writer to provide

training

for the V-Mail clients they chose to

use and to assist them in their tasks. We

recorded

qualitative

feedback by taking notes from interviews we conducted after

the training sessions as well as via questionnaires before and

after the two-week period. In addition, we saved each V-Mail

that they created and conﬁgured the V-Mail clients to log

all edits. After the two weeks, the artist shared the resulting

V-Mail in a regularly scheduled, “live” video-conference

with the

scientists

, gathering feedback on whether the story

covers important data features and whether it employs

appropriate storytelling techniques.

6.2 Observations and Results

We organize this section by the type of data we collected.

6.2.1 Pre-study Questionnaire

In the questionnaire we asked for the teams’ pre-existing

communications and related needs, and the two primary

participants ﬁlled out the questionnaire. Their responses

showed that they generally take different approaches for data

communication. For urgent matters, participants typically

communicate in-person or send e-mail-like messages. When

answering a question that could be answered in few words,

participants use an e-mail-like tool. However, when it takes

multiple steps to explain a concept, they typically use syn-

chronous communication, such as a live video conferencing

tool. The choice was the same for describing spatial features

in their data and for public-facing communication. When

asked about their use of visuals in e-mails, participants

responded differently. One does not use e-mails for sharing

3D visualization results. The other said that about 60% of her

e-mails contains images or videos, and 5% of these contain

annotations. With regards to device usage, both participants

typically have some access (2–8 hours) to high-end or

specialized devices. However, they typically use personal

PCs or mobile devices for science-related communications.

One participant indicated that her responses to questions

may vary depending on goals, as she often uses a high-end

PC when working on a video or visualization project.

6.2.2 Produced V-Mails

In total, the ﬁnal V-Mail contains ten data views (Fig. 12). The

ﬁrst three (#1–#3) provide an overview of data visualization,

highlighting all visible variables. The last shot (#10) is another

overview shot. Data views in the middle (#4–#9) describe two

different concepts about interactions of water masses and

currents on the ice shelf. Each concept was communicated

with three shots. The ﬁrst (#4, #7) gives a spatial context, the

second (#5, #8) zooms in further to focus on visual decisions,

and the last (#6, #9) zooms in further to relate the visual

decisions to speciﬁc data features. Each shot is discussed in

detail with textual comments contributed by both artists.

Interested in sharing their work to collaborating scien-

tists and possibly using it as video footage for their next

video projects, the artist requested another version of the

V-Mail that shows the ﬂy-through animation at a higher

resolution (1920

1080) and without the ﬂoating 3D pages.

Treating it like a ﬁnal draft of their work, the “ﬁnal” V-Mail

shows only the voice-over script, without all the low-level

communications such as questions in a form of comments.

6.2.3 Software Logs

Our log data and users’ comments made in the story indicate

that there were ﬁve back-and-forth communications over the

two weeks (Table 3). At each turn, after making edits, the

participants used their normal e-mail to pass the V-Mail link

to each other and request comments. Both URL and name

were used to reference updated V-Mails.

The textual comments were added to data views and

were used mainly to provide guidelines and script a voice-

over. Questions were also asked in a form of comments. For

example, confused with the scripting guidelines, the science

writer left questions on corresponding data views. We also

found that the comments contained imperfections, like typos

and this accidental use of caps lock, “hERE WE CAN SPEAK

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 11

Fig. 12. Visualizations of data views’ placements in the V-Mail participants created. Each circle represents a data view with a number indicating the

view’s order and with color indicating its thematic section.

TABLE 3

Five back-and-forth communications between participants.

# User Client Goals # of data # of edited # of views w/ # of views w/

views data views new comments new sketches

1 artist desktop capture data views, provide scripting guidelines 15 15 added 13 0

2 writer mobile draft a voice-over script, ask questions 15 none 9 0

3 artist desktop answer questions, make requests 15 none 3 0

4 writer mobile make requests 15 none 5 0

5 author desktop address the requests 10 5 removed, 0 0

1 rearranged

about.” Capitalization was also used for emphasis, as in

“IGNORE the above.” The artist and science writer did not

use the sketching feature to annotate data views.

The visual artist used the desktop client and the science

writer used the mobile client. While working on the script, the

science writer realized that a few data views were not needed.

While working on the mobile client, she left comments on

these extra data views to request a removal. Because the artist

using the desktop client forgot how to remove data views and

we (the authors) were also monitoring the communication,

we were able to quickly help out and resolve the request in

the desktop client.

6.2.4 Observations during introductory sessions

At the end of each training session, we asked users to now

begin to use the tool while we observed their actions and

answered any questions that came up.

In the ﬁrst V-Mail,

the artist

captured data views with

scripting guidelines written as comments (Table 3). The

installation steps and training time for this stage took about

20 minutes for the desktop-client. In the next hour session,

she created three V-Mail drafts from scratch (from an empty

storyboard) and did not report any major difﬁculties. She

discarded two of them, and proceeded to reﬁne to third by

adding a few more data views, rearranging their placement,

and adding notes and guidelines to send to her colleague.

The artist created V-Mails with multiple sections (i. e., data

views) to cover different data features (Fig. 12) and used

different approaches to extend and strengthen a story line

We noticed that the most recently added data view was often

used as a spatial reference frame for the next data view. For

example, after adding view #6, view #7 was described using

the language, “now looking down and reveal the currents.”

The participant also added data views “out of sequence” and

rearranged previously added views to strengthen the story.

In the second V-Mail,

the science writer

opened the

message from the artist, then proceeded to work on drafting

a voice-over script for each data view using the guidelines

contained in the V-Mail (Table 3). The science writer used the

mobile-client because it provided all features needed for her

tasks. The installation and training took her about 5 minutes.

She ﬂipped through pre-existing data views and added the

script as comments, ﬁnishing this task in next 30 minutes.

She did not report any major difﬁculties while using the tool.

6.2.5 Feedback from User Interviews

After they created their ﬁrst V-Mails, we interviewed the par-

ticipants to learn about their views of both tool and concept.

The artist described gaps in their current tools and

relevant cases where V-Mail could be most useful

. The

artist described her typical approach of creating a ﬂy-through

animation as, ﬁrst, establishing the concepts to convey and,

second, capturing the shots needed for the concepts. She

described the process as “melding story into visuals.” For

all but simple topics, in-person communication or video

conferencing are preferred for describing such visual and

spatial concepts, but in practice, they often resort to sending

several ﬂy-though animations to the scientists to conﬁrm the

concepts. In their traditional workﬂow, this process takes

a long time as they often require technical assistance from

additional team members to create their animations. V-Mails,

in contrast, enable artists, science writers, or other team

members to control this process on their own. They can focus

on identifying the right views to convey the concepts, and

the animation is automatically created. The ability to quickly

mock-up animations from data views expedites the creation

process, and the ability to view the results on mobile devices

and web browsers simpliﬁes the distribution.

Imagining how their experiences might translate to

future trial with the larger group

, the artist pointed out that

the mobile client will provide a means for the scientists to

provide feedback without the need to go through complex

installation steps and a long training. She also pointed out

the importance of the web-client as it enables them to reach

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 12

people outside of the group, e. g., the public and scientists

from other groups. The artist did not use

the sketching

feature to annotate data views

, even though, in several data

views, she described features at speciﬁc locations of the views

(e. g., eddies formed in a current shown in the bottom). When

asked about it during the interview, the artist explained that

she thought this would be useful in some situations, but in

this case she knew the science writer was already familiar

with data features and visual mappings from their close

recent work together.

The participants made several

feature requests

. She re-

quested to implement additional V-Mail clients for ParaView

[2], a visualization tool the team frequently uses. The science

writer suggested supporting audio recording to annotate

existing data views as it is often easier than writing thoughts

down. The writer told us that opening a V-Mail takes a long

time on her device. She suggested another data presentation

mode that shows a V-Mail without the animated transitions,

as this can reduce the download time and the animated

transitions might not be needed for some users who are

already familiar with data presented in the shared views.

Also, in our current implementation, the user ﬁrst needs to

add a data view before annotating it. She suggested another

approach where a data view is automatically added when

the user starts drawing on the current camera view.

6.2.6 Feedback from scientists

After the two weeks, the artist shared the resulting V-Mail in

a regularly scheduled, “live” video-conference with the larger

group, and led a discussion about the work using the web-

client. The scientists were most interested in seeing details

of water-mass distributions, current trails, and their impacts

on melting glaciers. The created V-Mail narrative covers

these regions of interests and employs proper storytelling

techniques—e. g., a series of ﬁrst overview, then zoomed-in

data views [33] connected with transitioning animations—

to show not only details of these regions but also their

contextual information. They were also interested in seeing

changes of speciﬁc regions over time. Unfortunately, time-

series data is not yet supported in the underlying visualiza-

tion software. The scientists requested this as a future work

and brainstormed ideas for storytelling techniques to convey

these in future iterations. They envisioned to station the

camera to focus on a speciﬁc region and then show changes

over time. They would do this for multiple data regions

and employ V-Mail’s interactive storytelling techniques to

return to previous camera locations to facilitate comparative

analysis of the sites.

6.2.7 Post-study Questionnaire

One user from the climate science team participated in the

questionnaire. Her responses reafﬁrm what she described

about our tool in the interview session (Section 6.2.5), giving

positive responses on features that enable her to make movies

or tell data stories. However, she noted several additional

points. The participant noted that connecting views with

animations is a crucial component in data communication.

The ability to pause in-between and play backwards as well

as the ability to see the 3D frames, however, received neutral

responses. Nonetheless, she was concerned about using small

mobile displays as they may not be able to convey the visual

detail of the visualizations.

6.3 Initial Feedback from a Brain Science Team

We integrated V-Mail into a different system that visualizes

crossing pathways in white matter areas in a human brain

(see Appendix B). We showed the V-Mail during a live video

conference to members of a neuroscience research team,

who was already familiar with the visualization system.

Their initial reaction was somewhat mixed; they told us

the tool itself is novel and could be useful in presenting

ﬁndings to a broader audience, but there is no urgent need

for such tool for their team-science collaboration. The team

already meets regularly, almost weekly, using a live video

conference or in a meeting room, and the team is led by

one senior member to whom his students report. They use

remote asynchronous communication tools such as e-mails

to coordinate the meetings or share datasets.

Participants described situations where cross-platform

collaboration tools like V-Mail may not work and could be

useful. In clinics, radiologists use specialized monitors for

viewing lossless DICOM images to see subtle differences in

tissue contrasts. Also a participant pointed out that there is a

limit to what information they can share with the public as

certain information is sensitive and private. In cases where

sharing information does not have such constraints, V-Mails

could be used to enhance the credibility of presented results.

They envision that stories created in V-Mail are submitted

as supplementary material for research papers. Linked from

ﬁgures or text, V-Mails could allow reviewers or readers to

validate the authors’ claims and further explore the data.

6.4 Discussion, Limitations, and Future Work

The success of adding a cross-device, asynchronous commu-

nication tool, like V-Mail, into the existing science team’s

workﬂow depends greatly on how the team already works.

The brain science team decided not to partake in a two-week

trial of V-Mail, as this team included just one senior member

and several grad students who already meet regularly to

discuss their work. They also raised concerns about security

with respect to patient data that should not be widely

shared (Section 6.3). The climate science team also expressed

a preference for synchronous communication. However,

schedules are more complex within this larger team, so the

ability to communicate asynchronously is more valuable.

In addition to using the comment feature for drafting a

voice-over script, participants used it to ask questions and

delegate tasks, and these comments often contain typos and

accidental use of caps lock. While these low-level communi-

cations were removed from the “ﬁnal” V-Mail, they can help

people to clarify tasks and enable other team members to

provide immediate help. All of these reinforce the chat-like

immediacy and draft-in-progress qualities of V-Mail. These

qualities are unusual in science communication, especially

high-end data visualization, but we believe they may, in fact,

help to put a human face on science communication, helping

scientists and stakeholders more freely discuss scientiﬁc data.

The science writer pointed out that communicating data

in a form of stories might introduce unnecessary overhead for

audiences who are already familiar with data (Section 6.2.5).

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 13

For these users, the ability to disable certain storytelling

features and share only portions of a data story could be

useful. By adapting the feature that encodes all viewing

properties in a shared link [29], we could associate an URL

parameter to each data view in a story, so that a speciﬁc

view could be referenced in e-mails or comments. The

feature would also be required if V-Mail is used with other

storytelling media, e. g., for providing interactive ﬁgures in a

paper submission as discussed by the brain science team. In

the same spirit, other storytelling paradigms that rely less on

animations could also be integrated in V-Mail. For instance,

scrolly-telling [34] can be used to rapidly control the speed

and direction of viewing. We could also spread out data

views in a layout, e. g., as done in data comics [5], which

could be more useful for audiences who are already familiar

with spatial locations of data features.

A major limitation of the current V-Mail implementation

is that it can only be integrated into applications built using

Unity3D, and a separate instance of the V-Mail plug-in is

needed for each system. To ease the integration process,

we are considering a more loosely coupled implementation

where a separate storyboard application communicates over

a network with visualization software. Since the webpage

and the mobile app do not rely on other data visualization

applications (these clients play a video), we can continue

using these clients for showing and interacting with stories

created from new connected software.

We observed that participants used lightweight clients if

the features required for their tasks were available in those

clients. We wonder, if the clients supported some lightweight

data exploration features, participants might have explored

data even though their tasks did not require this. Indeed, the

brain science team viewed facilitating interactive exploration

on the web-client as an important direction for future work.

Thus, upgrading the web-client with remote rendering [26] or

new web technologies [38] could be a useful next direction. In

addition, as mentioned by the artist and the brain scientists,

seeing visualization on small screens, like a phone, is just

limiting. Further studies are needed to better characterize

exactly how much value the capability provides to science

teams. In addition, further studies with more and other

domain experts should be carried out to conﬁrm our ﬁndings

from our qualitative evaluation.

7 CONCLUSION

On a conceptual level, our work extends the state of art

on storytelling in visualization and provides a rich tool

set for asynchronous remote collaboration. We build upon

existing spatial 3D data storytelling techniques that tightly

couple storytelling with data exploration and employ semi-

automatic procedures to expedite the story creation process.

What is unique in V-Mail is the ability to collectively work

on data stories, which is often seen with abstract data

visualization, e. g., via web-based visualizations, but less so

with visualization of spatial 3D data. Essentially this multi-

author editing in V-Mail is a type of parallel storytelling,

with the difference that users not only view or edit shared

data stories but also do this across different devices. V-Mail

is thus an example of using multiple display devices in

collaborative data visualization [15], [16]. V-Mail focuses,

however, on asynchronous remote communication and pro-

vides lightweight mobile and web clients for partaking in

collaboration and simpliﬁes the distribution process using

URLs to reference data stories.

The desktop-client is an example of asynchronous col-

laboration frameworks that support collaborative activities

around exploratory visualizations. The mobile-client, how-

ever, supports these activities around video-based visualiza-

tions created from the desktop-client. The ability to provide

visual feedback directly onto shared visualization results is

similar to the ability to annotate on shared data views and

stories in the web-based visualization. However, there are

two major differences in our approach: (1) communication

mechanisms are tailored for spatial 3D data visualization,

e. g., with spatial annotations and animated transitions,

and (2) the app is designed to solely work with shared

visualization results. The latter point may be of interest to

visualization practitioners as this approach enables them

to bypass installation steps and computation requirements

often imposed by using exploratory visualization software

such as ParaView [2]. Also this separation suggests an inter-

esting future direction of turning V-Mail into a collaborative

framework that uniﬁes well-known software in the ﬁeld for

creating data stories and conducting discussion.

In our ﬁrst V-Mail deployment and evaluation, we took

the approach of asking a real science team to integrate V-Mail

in whatever ways they found most useful for the tasks at

hand during a two-week period. The results were generally

positive, re-conﬁrming the urgent need for improved tools for

spatial communication in this context, while also suggesting

areas of required improvements to advance V-Mail from

the prototype stage to a production-level tool. In this ﬁrst

application, only the artist and science writer actively used

the tool to draft a video-based visualization, and the scientists

were involved at the last stage to provide feedback on the

created visualization. The brain science team provided initial

feedback on the tool but did not partake in a longitude

study. Notably, if we are able to secure funding to support

additional work on V-Mail, we plan follow-on studies to

understand how V-Mail might be used and/or tailored to

best support speciﬁc scientist-to-scientist, public-to-science-

team, and lay-person-to-specialist communications.

ACKNOWLEDGMENTS

The authors thank Seth Johnson and Bridger Herman for

assistance with integrating the ABR engine. This research

was supported in part by the National Science Foundation

(IIS-1704604, IIS-1704904).

IMAGES/FIGURES LICENSE AND COPYRIGHT

We as authors state that all of our ﬁgures in this article are

and remain under our own personal copyright, with the

permission to be used here. We also make them available

under the Creative Commons Attribution 4.0 International

(c b CC BY 4.0) license and share them at osf.io/qehvs.

REFERENCES

[1]

D. Acevedo, E. Vote, D. H. Laidlaw, and M. S. Joukowsky,

“Archaeological data visualization in VR: Analysis of lamp ﬁnds at

the Great Temple of Petra, a case study,” in Proc. Visualization. Los

Alamitos: IEEE CS, 2001, pp. 493–496. doi:

10.1109/VISUAL.2001.964560

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 14

[2]

J. Ahrens, B. Geveci, and C. Law, “Paraview: An end-user tool

for large data visualization,” in The Visualization Handbook, C. D.

Hansen and C. R. Johnson, Eds. Oxford: Elsevier, 2005, ch. 36, pp.

717–731. doi: 10.1016/B978-012387582-2/50038-1

[3]

H. Akiba, C. Wang, and K.-L. Ma, “AniViz: A template-based

animation tool for volume visualization,” IEEE Comput Graph Appl,

vol. 30, no. 5, pp. 61–71, 2010. doi: 10.1109/MCG.2009.107

[4]

B. Bach, N. Kerracher, K. W. Hall, S. Carpendale, J. Kennedy, and

N. Henry Riche, “Telling stories about dynamic networks with

graph comics,” in Proc. CHI. New York: ACM, 2016, pp. 3670–

3682. doi: 10.1145/2858036.2858387

[5]

B. Bach, Z. Wang, M. Farinella, D. Murray-Rust, and

N. Henry Riche, “Design patterns for data comics,” in Proc. CHI.

New York: ACM, 2018, pp. 38:1–38:12. doi: 10 .1145/3173574.3173612

[6]

K. Bladin, E. Axelsson, E. Broberg, C. Emmart, P. Ljung, A. Bock,

and A. Ynnerman, “Globe browsing: Contextualized spatio-

temporal planetary surface visualization,” IEEE Trans Vis Comput

Graph, vol. 24, no. 1, pp. 802–811, 2018. doi:

10.1109/TVCG.2017.2743958

[7]

P. de Lange, P. Nicolaescu, J. Benscheid, and R. Klamma, “Collab-

orative non-linear storytelling around 3D objects,” in Proc. ICWL.

Cham: Springer, 2018, pp. 88–98. doi: 10 .1007/978-3-319-96565-9 9

[8]

N. Duarte, Data Story: Explain Data and Inspire Action Through Story.

Ideapress Publishing, 2019.

[9]

R. Eccles, T. Kapler, R. Harper, and W. Wright, “Stories in geotime,”

Inf Vis, vol. 7, no. 1, pp. 3–17, 2008. doi: 10.1057/palgrave.ivs.9500173

[10]

N. Gershon and W. Page, “What storytelling can do for information

visualization,” Commun ACM, vol. 44, no. 8, pp. 31–37, 2001. doi:

10.1145/381641.381653

[11]

J. Heer and M. Agrawala, “Design considerations for collaborative

visual analytics,” Inf Vis, vol. 7, no. 1, pp. 49–62, 2008. doi:

10.

1057/palgrave.ivs.9500167

[12]

J. Heer and G. Robertson, “Animated transitions in statistical data

graphics,” IEEE Trans Vis Comput Graph, vol. 13, no. 6, pp. 1240–

1247, 2007. doi: 10.1109/TVCG.2007.70539

[13]

J. Heer, F. B. Vi

egas, and M. Wattenberg, “Voyagers and voyeurs:

supporting asynchronous collaborative information visualization,”

in Proc. CHI. New York: ACM, 2007, pp. 1029–1038. doi:

10.

1145/1240624.1240781

[14]

S. Iserhardt-Bauer, P. Hastreiter, T. Ertl, K. Eberhardt, and

B. Tomandl, “Case study: Medical Web service for the automatic 3D

documentation for neuroradiological diagnosis,” in Proc. Visualiza-

tion. Los Alamitos: IEEE CS, 2001, pp. 425–581. doi:

10.1109/VISUAL.

2001.964542

[15]

D. H. Jeong, S.-Y. Ji, E. A. Suma, B. Yu, and R. Chang, “Designing a

collaborative visual analytics system to support users’ continuous

analytical processes,” Hum-Centric Comput Inf Sci, vol. 5, no. 1, pp.

5:1–5:20, 2015. doi: 10.1186/s13673-015-0023-4

[16]

D. H. Jeong, E. Suma, T. Butkiewicz, W. Ribarsky, and R. Chang,

“A continuous analysis process between desktop and collaborative

visual analytics environments,” in Proc. VAST. Los Alamitos: IEEE

CS, 2010, pp. 231–232. doi: 10.1109/VAST.2010.5652958

[17]

S. Johnson, F. Samsel, G. Abram, D. Olson, A. J. Solis, B. Herman,

P. J. Wolfram, C. Lenglet, and D. F. Keefe, “Artifact-based rendering:

Harnessing natural and traditional visual media for more expres-

sive and engaging 3D visualizations,” IEEE Trans Vis Comput Graph,

vol. 26, no. 1, pp. 492–502, 2019. doi: 10.1109/TVCG.2019.2934260

[18]

T. Klein, L. Autin, B. Kozl

ıkov

a, D. S. Goodsell, A. Olson, M. E.

oller, and I. Viola, “Instant construction and visualization of

crowded biological environments,” IEEE Trans Vis Comput Graph,

vol. 24, no. 1, pp. 862–872, 2017. doi: 10.1109/TVCG.2017.2744258

[19]

C. N. Knaﬂic, Storytelling with Data: A Data Visualization Guide for

business Professionals. Hoboken: Wiley, 2015.

[20]

R. Kosara and J. Mackinlay, “Storytelling: The next step for

visualization,” Comput, vol. 46, no. 5, pp. 44–50, 2013. doi:

10.1109/MC.

2013.36

[21]

D. Kou

ril, O. Strnad, P. Mindek, S. Halladjian, T. Isenberg, M. E.

oller, and I. Viola, “Molecumentary: Adaptable narrated docu-

mentaries using molecular visualization,” IEEE Trans Vis Comput

Graph, 2022, to appear. doi: 10.1109/TVCG.2021.3130670

[22]

B. Lee, R. H. Kazi, and G. Smith, “SketchStory: Telling more

engaging stories with data through freeform sketching,” IEEE

Trans Vis Comput Graph, vol. 19, no. 12, pp. 2416–2425, 2013. doi:

10.

1109/TVCG.2013.191

[23]

B. Lee, N. H. Riche, P. Isenberg, and S. Carpendale, “More than

telling a story: Transforming data into visually shared stories,”

IEEE Comput Graph Appl, vol. 35, no. 5, pp. 84–90, 2015. doi:

10.

1109/MCG.2015.99

[24]

I. Liao, W.-H. Hsu, and K.-L. Ma, “Storytelling via navigation: A

novel approach to animation for scientiﬁc visualization,” in Proc.

Smart Graphics. Cham: Springer, 2014, pp. 1–14. doi:

10.1007/978-3

-319-11650-1 1

[25]

E. M. Lidal, M. Natali, D. Patel, H. Hauser, and I. Viola, “Geological

storytelling,” Comput Graph, vol. 37, no. 5, pp. 445–459, 2013. doi:

10.1016/j.cag.2013.01.010

[26]

E. J. Luke and C. D. Hansen, “Semotus Visum: A ﬂexible remote

visualization framework,” in Proc. Visualization. Los Alamitos:

IEEE CS, 2002, pp. 61–68. doi: 10.1109/VISUAL.2002.1183758

[27]

P. Lundblad and M. Jern, “Geovisual analytics and storytelling

using HTML5,” in Proc. IV. Los Alamitos: IEEE CS, 2013, pp.

263–271. doi: 10.1109/IV.2013.35

[28]

K.-L. Ma, I. Liao, J. Frazier, H. Hauser, and H.-N. Kostis, “Scientiﬁc

storytelling using visualization,” IEEE Comput Graph Appl, vol. 32,

no. 1, pp. 12–19, 2012. doi: 10.1109/MCG.2012.24

[29]

J. Maitin-Shepard, A. Baden, W. Silversmith, E. Perlman, F. Collman,

T. Blakely, J. Funke, C. Jordan, B. Falk, N. Kemnitz, tingzhao,

C. Roat, M. Castro, S. Jagannathan, moenigin, J. Clements, A. Hoag,

B. Katz, D. Parsons, J. Wu, L. Kamentsky, P. Chervakov, P. Hubbard,

S. Berg, J. Hoffer, A. Halageri, C. Machacek, K. Mader, L. Roeder,

and P. H. Li, “google/neuroglancer:,” Software: WebGL-based

viewer for volumetric data, 2021. doi: 10.5281/zenodo.5573294

[30]

M. Meuschke, L. Garrison, N. Smit, S. Bruckner, K. Lawonn, and

B. Preim, “Towards narrative medical visualization,” arXiv preprint

2108.05462, 2021.

[31]

N. H. Riche, C. Hurter, N. Diakopoulos, and S. Carpendale, Data-

Driven Storytelling. Boca Raton: CRC Press, 2018. doi:

10.1201/

9781315281575

[32]

E. Segel and J. Heer, “Narrative visualization: Telling stories with

data,” IEEE Trans Vis Comput Graph, vol. 16, no. 6, pp. 1139–1148,

2010. doi: 10.1109/TVCG.2010.179

[33]

B. Shneiderman, “The eyes have it: A task by data type taxonomy

for information visualizations,” in Proc. VL. Los Alamitos: IEEE

CS, 1996, pp. 336–343. doi: 10.1109/VL.1996.545307

[34]

C. D. Stolper, B. Lee, N. H. Riche, and J. Stasko, “Emerging and

recurring data-driven storytelling techniques: Analysis of a curated

collection of recent stories,” Microsoft Research, Washington, Tech.

Rep. MSR-TR-2016-14, 2016.

[35]

M. Th

ony, R. Schn

urer, R. Sieber, L. Hurni, and R. Pajarola,

“Storytelling in interactive 3D geographic visualization systems,”

ISPRS Int J Geo-Inf , vol. 7, no. 3, pp. 123:1–123:14, 2018. doi:

10.

3390/ijgi7030123

[36]

C. Tong, R. Roberts, R. Borgo, S. Walton, R. Laramee, K. Wegba,

A. Lu, Y. Wang, H. Qu, Q. Luo et al., “Storytelling and visualization:

An extended survey,” Inf , vol. 9, no. 3, pp. 65:1–65:42, 2018. doi:

10.

3390/info9030065

[37]

M. Tsang, G. W. Fitzmaurice, G. Kurtenbach, A. Khan, and B. Bux-

ton, “Boom Chameleon: Simultaneous capture of 3D viewpoint,

voice and gesture annotations on a spatially-aware display,” in

Proc. UIST. New York: ACM, 2002, pp. 111–120. doi:

10.1145/571985.

572001

[38]

W. Usher and V. Pascucci, “Interactive visualization of terascale

data in the browser: Fact or ﬁction?” in Proc. LDAV. Los Alamitos:

IEEE CS, 2020, pp. 27–36.

[39]

F. B. Viegas, M. Wattenberg, F. Van Ham, J. Kriss, and M. McKeon,

“ManyEyes: A site for visualization at internet scale,” IEEE Trans Vis

Comput Graph, vol. 13, no. 6, pp. 1121–1128, 2007. doi:

10.1109/TVCG.

2007.70577

[40]

M. Wohlfart and H. Hauser, “Story telling for presentation in

volume visualization,” in Proc. EuroVis. Goslar: Eurographics

Assoc, 2007, pp. 91–98. doi: 10.2312/VisSym/EuroVis07/091-098

[41]

K. Wongsuphasawat, D. Smilkov, J. Wexler, J. Wilson, D. Mane,

D. Fritz, D. Krishnan, F. B. Vi

egas, and M. Wattenberg, “Visualizing

dataﬂow graphs of deep learning models in tensorﬂow,” IEEE Trans

Vis Comput Graph, vol. 24, no. 1, pp. 1–12, 2017. doi:

10.1109/TVCG.

2017.2744878

[42]

A. Ynnerman, J. L

owgren, and L. Tibell, “Exploranation: A new

science communication paradigm,” IEEE Comput Graph Appl,

vol. 38, no. 3, pp. 13–20, 2018. doi: 10.1109/MCG.2018.032421649

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.

SUBMISSION TO IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 15

Jung Who Nam

is a doctoral student at the Uni-

versity of Minnesota, Twin Cities. His research in-

terests include scientiﬁc visualization, immersive

analytics, and data storytelling. He is particularly

interested in novel visualization and interaction

methods that support exploration and collabora-

tion around spatial 3D data visualization.

Tobias Isenberg

is a senior research scientist

at Inria, France. Previously he held positions

as post-doctoral fellow at the University of Cal-

gary, Canada, and as assistant professor at the

University of Groningen, the Netherlands. His

research interests include scientiﬁc visualiza-

tion, illustrative and non-photorealistic rendering,

and interactive visualization techniques. He is

particularly interested in interactive visualization

environments for 3D spatial data.

Daniel F. Keefe

is a Distinguished University

Teaching Professor and Professor of Computer

Science and Engineering at the University of

Minnesota. His research centers on scientiﬁc

data visualization and interactive computer graph-

ics. In addition to his work in computer science,

Keefe is also an accomplished artist and has

published and exhibited work in top international

venues for digital art. Before joining the University

of Minnesota, Keefe did post-doctoral work at

Brown University jointly with the departments of

Computer Science and Ecology and Evolutionary Biology and with the

Rhode Island School of Design. He received the Ph.D. in 2007 from

Brown University’s Department of Computer Science and the B.S. in

Computer Engineering summa cum laude from Tufts University in 1999.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited and

content may change prior to final publication. Citation information: DOI 10.1109/TVCG.2022.3229017

Authorized licensed use limited to: Hong Kong University of Science and Technology. Downloaded on July 19,2023 at 08:38:41 UTC from IEEE Xplore. Restrictions apply.